1. Technical Field
The present invention relates to software testing/verification and more particularly to systems and methods for testing software that employ testing with machine learning to debug programs.
2. Description of the Related Art
Bug finding approaches can be divided into two categories: dynamic and static techniques and combinations thereof. Dynamic testing and static analysis are commonly used bug finding techniques at the extreme ends of this spectrum, dynamic testing being the more commonly used. Testing is performed by running or simulating the system under many user-provided test cases, which define program inputs and the runtime environment. Static analysis uses symbolic reasoning techniques to find bugs in software. Both techniques can be performed in numerous ways based on the program under test and the testing requirements. For example, unit testing/local static analysis is used to verify and sometimes derive contracts at function interfaces whereas global testing and analysis aim to analyze the entire program as a whole to identify bugs.
Testing begins with the specification of test cases and the runtime environment, and aims to find bugs by executing the program. Test case generation is a well studied problem for different types of systems with a host of techniques ranging from the informal to the formal. The main advantage of testing is the reliability of the test information especially when the testing is performed for the entire program. In practice, however, achieving an acceptable coverage for large software systems is prohibitive.
As a result, testing is rarely applied exhaustively except for safety critical software. To complicate matters, the notion of coverage is not well defined and at times the correlation between a particular notion of coverage such as line coverage and the number of unexplored bugs is weak. Static Analysis uses reasoning techniques to deduce the presence of bugs or prove their absence. In theory, static analysis techniques are sound and exhaustive, i.e., they guarantee 100% coverage of the program paths and input values.
In practice, soundness and exhaustive searches are sacrificed for scalability. Nevertheless, the success of the many popular static analysis tools such as CoVerity™, FindBugs™, PreFix™, PolySpace™ and so on are mainly due to their independence from an actual running environment and vastly improved code coverage. One key disadvantage is the intractability of static analysis techniques and the high false positive rates.